Skip to main content

AI and Machine Learning

Bluefin's developer experience fully supports local AI development workflows. GPU Acceleration for both Nvidia and AMD are included out of the box and usually do not require any extra setup.

AI is an extension to cloud native

Bluefin's focus in AI is providing a generic API endpoint to the operating system that is controlled by the user. Similar to how Bluefin ships podman, we feel that ramalama and other great open source tools can be used for many useful purposes.

AI Lab with Podman Desktop​

The AI Lab extension can be installed inside the included Podman Desktop to provide a graphical interface for managing local models:

image

Ramalama​

Ramalama is included to manage local models and is the prefered default experience. It's for people who work with local models frequently and need advanced features. It offers the ability to pull models from huggingface, ollama, and any container registry. By default it pulls from ollama.com, check the Ramalama documentation for more information.

Ramalama's command line experience is similar to Podman, examples include:

ramalama pull llama3.2:latest
ramalama run llama3.2  
ramalama run deepseek-r1

You can also serve the models locally:

ramalama serve deepseek-r1

Then go to http://127.0.0.0:8080 in your browser.

Ramalama will automatically pull in anything your host needs to do the workload. The images are also stored in the same container storage as your other containers. This allows for centralized management of the ai models and other podman images:

❯ podman images
REPOSITORY                                 TAG         IMAGE ID      CREATED        SIZE
quay.io/ramalama/rocm                      latest      8875feffdb87  5 days ago     6.92 GB

Alpaca Graphical Client​

For light chatbot usage we recommend that users install Alpaca to manage and chat with your LLM models from within a native desktop application. Alpaca supports Nvidia and AMD1 acceleration natively and includes ollama.

image

Running Ollama as a Service​

Ollama can also be used for people who prefer to use that tool. If you want third party tools to integrate with it, (for example an IDE) you should consider installing it in a docker container.

To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin) with:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then, choose a folder where to install the ollama container (for example ~/Containers/ollama) and inside it create a new file named docker-compose.yaml with the following content:

---
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    ports:
      - 11434:11434
    volumes:
      - ./ollama_v:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities:
                - gpu

Finally, open a terminal in the folder containing the file just created and start the container with

docker compose up -d

and your ollama instance should be up and running at http://127.0.0.1:11434!

NOTE: if you still want to use Alpaca as one of the way of interacting with Ollama, you can open the application, then go to Preferences, toggle the option Use the Remote Connection to Ollama, specify the endpoint above (http://127.0.0.1:11434) as Server URL (leave Bearer Token empty) in the dialog that will pop up and then press Connect. This way you should be able to manage the models installed on your ollama container and chat with them from the Alpaca GUI.

Footnotes​

  1. For proper AMD support, the Flatpak extension com.jeffser.Alpaca.Plugins.AMD must also be installed. ↩